1. Field of the Invention
The present invention relates to an image processing device, method, and program, for generating a color image from a sparsely color sampled image which is acquired by using a single-chip color image sensor, and further relates to an image processing device, method, and program, for performing a demosaicing process including processing such as noise reduction, optical low-pass filter compensation and pixel drop-out processing.
2. Description of the Related Art
Hitherto, single-CCD (charge-coupled device) color image sensors have been widely used in color imaging devices such as digital cameras and compact video cameras. A single-CCD color image sensor includes, on the front face of a solid-state image sensor, a color filter array in which color transmission properties that differ for each pixel are provided in a specified array or a random array. For example, a color filter array such as illustrated in FIG. 4 uses an RGB (Red-Green-Blue) color Bayer pattern in which the color filters with color transmission characteristics corresponding to R, G, B are cyclically ordered every 2×2 pixels at a ratio of R:G:B=1:2:1 (see U.S. Pat. No. 3,971,065).
The output image of such a single-CCD color image sensor (RAW image) is a so-called color mosaic image in which each pixel only has information about a single color, and therefore as a color imaging device, a demosaicing process which converts the RGB color information (color information) as to the various pixels of this color mosaic image into a color image becomes necessary.
With the demosaicing process, a method which spatially interpolates the color information lacking in various pixels from the surrounding pixels is widely used.
However, with the aforementioned RGB color Bayer pattern, for example, simply interpolating results in the resolution essentially decreasing, because G has only one half of all the pixels and R and B each have only one fourth of all the pixels.
In order to solve this problem, an interpolation method using applicable directional selective smoothing has been proposed, such as a two-directional correlation method (see U.S. Pat. No. 4,663,655) or an eight-direction edge detecting method (for example, see Chang, Ed. et al., “Color Filter Array Recovery Using a Threshold-based Variable Number of Gradients” Proceedings of SPIE, January, 1999).
On the other hand, a recent trend has been to emphasize not only pixel count, but also color reproducibility and dynamic ranging as capabilities of a digital camera. Thus, a color filter array having color filters of four or more colors may be used (see Japanese Patent Laid-Open No. 2003-284084), or a color filter array having color filters of six or more types of colors may be used by providing variation in magnitude of transmittance even when the chrominance is similar (for example, see U.S. Published patent application No. US 2003-160875AA). Further, in order to suppress color moiré, a technique has been proposed using a random color array in the color filter array or a cycle longer than 2×2 pixels (see Japanese Patent Laid-Open No. 9-168157).
Further, with iterative restoration in which a decompressed image which is similar to the source image is obtained by repeating interpolation calculations, a method in which inappropriate image restoration which restores up to a bandwidth past the Nyquist frequency can be avoided, by considering the birefringence of the optical low-pass filter which is placed between the photography optical system and the image sensor and by limiting the frequency band of the restored image, has been proposed (see “A Study of Demosaicing With Regard to Characteristics of a Birefringent Optical Low-Pass Filter” by Takashi Komatsu and Takahiro Saito, 2003 Annual Conference, Institute of Image Formation and Television Engineers).
However, the demosaicing processes proposed therein are based on spatial interpolation of same color pixels, and therefore when the number of colors in the color filter is increased, the pixel count for a specified color decreases and spatial resolution degrades and becomes a problem. Further, even with a random color filter array, similar problems can occur because regions wherein pixels of the same color become locally non-dense occur.
Further, the discrepancy of the color permeability characteristics, based on manufacturing error of the color filter array between the pixels of the same pixel color, is manifested as noise.